Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ISSUE-363: Add the ability to Drop columns with DoricColumns #366

Merged
merged 4 commits into from
Jul 18, 2023

Conversation

Amalicia
Copy link
Contributor

@Amalicia Amalicia commented Jun 23, 2023

Description

This PR updates transform ops to add the drop column found in Spark. The doric function has the following extra functionality:

  • It will return an error if the column is not found in the DataFrame
  • You can drop multiple columns using the col syntax

Related Issue and dependencies

How Has This Been Tested?

Unit tests have been updated to include testing of both drop functions.

  • This pull request contains appropriate tests?:
    • YES

@Amalicia Amalicia requested a review from a team as a code owner June 23, 2023 14:29
@github-actions github-actions bot added spark_2.4 PR changes to spark 2.4 spark_3.0 PR changes to spark 3.0 spark_3.1 PR changes to spark 3.1 spark_3.2 PR changes to spark 3.2 spark_3.3 PR changes to spark 3.3 spark_3.4 PR changes to spark 3.4 labels Jun 23, 2023
@github-actions
Copy link

github-actions bot commented Jun 23, 2023

:octocat: This is an auto-generated comment created by:

  • Date : 2023-07-18 11:12:01 +0000 (UTC)
  • Workflow : PR comment
  • Job name : create_test_summary_report
  • Run : 5586690783
  • Commit : 9943db6 ISSUE-363: Add additional tests
Actor Triggering actor Sender
eruizalo
eruizalo
eruizalo
eruizalo
eruizalo
eruizalo
Triggered by:

Test summary report 📊

Spark version testing
2.4.1 633 passed, 2 skipped
2.4.2 633 passed, 2 skipped
2.4.3 633 passed, 2 skipped
2.4.4 633 passed, 2 skipped
2.4.5 633 passed, 2 skipped
2.4.6 634 passed, 2 skipped
2.4.7 634 passed, 2 skipped
2.4 634 passed, 2 skipped
3.0.0 684 passed, 2 skipped
3.0.1 684 passed, 2 skipped
3.0.2 684 passed, 2 skipped
3.0 684 passed, 2 skipped
3.1.0 712 passed, 2 skipped
3.1.1 712 passed, 2 skipped
3.1.2 712 passed, 2 skipped
3.1 712 passed, 2 skipped
3.2.0 716 passed, 2 skipped
3.2.1 716 passed, 2 skipped
3.2.2 716 passed, 2 skipped
3.2 716 passed, 2 skipped
3.3.0 716 passed, 2 skipped
3.3.1 716 passed, 2 skipped
3.3 716 passed, 2 skipped
3.4 716 passed, 2 skipped

@codecov
Copy link

codecov bot commented Jun 23, 2023

Codecov Report

Merging #366 (80af2da) into main (446d49d) will increase coverage by 0.01%.
The diff coverage is 100.00%.

❗ Current head 80af2da differs from pull request most recent head 9943db6. Consider uploading reports for the commit 9943db6 to get more accurate results

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #366      +/-   ##
==========================================
+ Coverage   97.61%   97.61%   +0.01%     
==========================================
  Files          62       62              
  Lines        1253     1256       +3     
  Branches       20       17       -3     
==========================================
+ Hits         1223     1226       +3     
  Misses         30       30              
Flag Coverage Δ
spark-2.4.x 94.70% <100.00%> (+0.01%) ⬆️
spark-3.0.x 96.75% <100.00%> (+0.01%) ⬆️
spark-3.1.x 97.50% <100.00%> (+0.01%) ⬆️
spark-3.2.x 97.71% <100.00%> (+0.01%) ⬆️
spark-3.3.x 97.71% <100.00%> (+0.01%) ⬆️
spark-3.4.x 97.71% <100.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
core/src/main/scala/doric/sem/TransformOps.scala 100.00% <100.00%> (ø)

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 446d49d...9943db6. Read the comment docs.

@eruizalo eruizalo requested a review from alfonsorr July 18, 2023 11:15
@eruizalo eruizalo merged commit c29f531 into hablapps:main Jul 18, 2023
55 checks passed
@Amalicia Amalicia deleted the feature/ISSUE-363 branch July 19, 2023 12:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
spark_2.4 PR changes to spark 2.4 spark_3.0 PR changes to spark 3.0 spark_3.1 PR changes to spark 3.1 spark_3.2 PR changes to spark 3.2 spark_3.3 PR changes to spark 3.3 spark_3.4 PR changes to spark 3.4
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature request]: Add the ability to use the drop function with doric columns
3 participants